Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

module commands to be marked as unstable and require clients to opt in #2766

Closed
wants to merge 72 commits into from

Conversation

sjpotter
Copy link
Contributor

infrastrucutre code to enable search module commands to be marked unstable in RESP3 client usage.

needs, finish conversion of commands to v5, where I'm stuck at the moment

@sjpotter sjpotter requested a review from leibale May 26, 2024 11:09
return new ((this as any).Multi as Multi)(
this._executeMulti.bind(this),
this._executePipeline.bind(this)
this._executePipeline.bind(this),
this._self.#options
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multi should throw an error on attempted executoin of an unstableResp3 command. this enables that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, with what we discussed yesterday, putting the handler on object via attachCommand, this is no longer neccessary, so reverted.

@@ -163,6 +167,9 @@ export default class RedisClient<
static #createModuleCommand(command: Command, resp: RespVersions) {
const transformReply = getTransformReply(command, resp);
return async function (this: NamespaceProxyClient, ...args: Array<unknown>) {
if (command.unstableResp3Module && resp == 3 && !this._self._self.#options?.unstableResp3Modules) {
throw new Error("unstable resp3 module, client not configured with proper flag");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to update the message + include a link

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, but unsure what it should be at the moment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a FIXME to it, to know we have to work on that string

@@ -163,6 +167,9 @@ export default class RedisClient<
static #createModuleCommand(command: Command, resp: RespVersions) {
const transformReply = getTransformReply(command, resp);
return async function (this: NamespaceProxyClient, ...args: Array<unknown>) {
if (command.unstableResp3Module && resp == 3 && !this._self._self.#options?.unstableResp3Modules) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move the if + throw to a function and reuse it everywhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. makes sense. would be a function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something like

export function resp3UnstableCheck(resp: RespVersions, unstableResp3Command?: boolean, unstableResp3Client?: boolean) {
  if (resp == 3 && unstableResp3Command && !unstableResp3Client) {
    throw new Error("unstable resp3 module, client not configured with proper flag");
  }
}```

and usage ala

  resp3UnstableCheck(resp, command.unstableResp3Module, this._self._self.#options?.unstableResp3Modules);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed on zoom, moved this to how the command is actually attached to object in attachCommand

@@ -2,6 +2,7 @@ import COMMANDS from '../commands';
import RedisMultiCommand, { MULTI_REPLY, MultiReply, MultiReplyType, RedisMultiQueuedCommand } from '../multi-command';
import { ReplyWithTypeMapping, CommandReply, Command, CommandArguments, CommanderConfig, RedisFunctions, RedisModules, RedisScripts, RespVersions, TransformReply, RedisScript, RedisFunction, TypeMapping } from '../RESP/types';
import { attachConfig, functionArgumentsPrefix, getTransformReply } from '../commander';
import { RedisClientOptions } from '.';

type CommandSignature<
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be "never" when using RESP3 without enabling unstable commands..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont know how to do that into the type (I wasn't eager to extend the typing of the client even further)

@sjpotter sjpotter force-pushed the v5-search-broken branch from 36ad751 to 2b6e99a Compare June 2, 2024 13:21
sjpotter added 7 commits June 3, 2024 17:39
update unstableResp3Module flag to only commands that have unstable api

add ability to tag commands to ignore user type mapping (as transformReply will convert it to a non typed map type)
	update bloom info commands to work with new non typed map ability

fill in search/time-series commands and tag with unstableResp3Modules flag as appropriate
@sjpotter sjpotter force-pushed the v5-search-broken branch from 4fa8734 to f13f47a Compare June 3, 2024 14:39
// TODO: ?!??!
3: undefined as unknown as () => any
}
3: undefined as unknown as () => ArrayReply<NullReply | BlobStringReply | ArrayReply<BlobStringReply>>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too sure if this is right..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in resp3, the same reply that is possible in resp2 is returned in resp3, just wrapped in an array.

pub fn json_type<M: Manager>(manager: M, ctx: &Context, args: Vec<RedisString>) -> RedisResult {
    let mut args = args.into_iter().skip(1);
    let key = args.next_arg()?;
    let path = Path::new(args.next_str().unwrap_or(default_path()));

    let key = manager.open_key_read(ctx, &key)?;

    let value = if path.is_legacy() {
        json_type_legacy::<M>(&key, path.get_path())?
    } else {
        json_type_impl::<M>(&key, path.get_path())?
    };

    // Check context flags to see if RESP3 is enabled and return the appropriate result
    if is_resp3(ctx) {
        Ok(vec![value].into())
    } else {
        Ok(value)
    }
}

so if one is ok with the resp2, response, the resp3 response looks right to me.

the problem is really in me messing up the conversion in v4, we have

export declare function transformReply(): string | null | Array<string | null>;

so what we need is the ArrayReply<BlobStringReply | NullReply>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made the change I mentioned now

@@ -3,8 +3,8 @@ import { SimpleStringReply, Command, RedisArgument } from '@redis/client/dist/li
export default {
FIRST_KEY_INDEX: undefined,
IS_READ_ONLY: true,
transformArguments(index: RedisArgument, cursorId: RedisArgument) {
return ['FT.CURSOR', 'DEL', index, cursorId];
transformArguments(index: RedisArgument, cursorId: number) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cursorId can go outside the float64 range.. string has to be supported, number is nice to have

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, was a copy form v4 which used number. trying to remember why I changed it from what you did originally, I suspect number might be used elsewhere and needs to be fixed, will look into that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I'm also unconvinced that "number" is nice to have, its a "number", but in practice, cursors should be opaque, and we shouldn't encourage users to think of it as a number, so thinking that RedisArgument as is, should be enough (as you had originally)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the issue here is really due to AGGREGATE_WITHCURSOR (where cursor id is returned to user). Changed it to be a NumberReply there. I think before I ran into some problems with tests, will have to redo and see what the problem was.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i.e. the problem is that AGGREGATE_WITHCURSOR when cursor is a NumberReply will default to returning as a number.

instead returning NumberReply (from WITHCURSOR) for cursor and taking an UnwrapReply for cursor in cursor read/del and its up to the user to therefore have a proper number conversion setup.

Honestly, not happy with it. I'd want to force the NumberReply to be treated as a string and not as a number in WITHCURSOR (and then can force a string in cursor functions)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"I'd want to force the NumberReply to be treated as a string and not as a number in WITHCURSOR (and then can force a string in cursor functions)" - I'd want the server to reply with a string and not a number..
The server not gonna change, we should probably support both.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we have the NumberReply that can be both. With that said, I'd think we should document this (i.e. we strongly reccomend one overrides the type mapping for NumberReplies when using the WITHCURSOR function).

A better solution (which I dont know how to implement) would be to be able to tag commands with a default type mapping that can only override default (blank) client type mapping, but not type mapping that the user specifies.

i.e. by default we might say a NumberReply maps to a a js number, but for some commands we can say it maps to string, and the client would then just do the right thing by default always without user intervention required.

@@ -8,14 +8,15 @@ export interface FtCursorReadOptions {
export default {
FIRST_KEY_INDEX: undefined,
IS_READ_ONLY: true,
transformArguments(index: RedisArgument, cursor: RedisArgument, options?: FtCursorReadOptions) {
const args = ['FT.CURSOR', 'READ', index, cursor];
transformArguments(index: RedisArgument, cursor: number, options?: FtCursorReadOptions) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same cursor comment

@@ -1,28 +1,28 @@
// import { RedisArgument, SimpleStringReply, Command } from '@redis/client/dist/lib/RESP/types';
// import { Params, pushParamsArgs } from ".";
import { RedisArgument, SimpleStringReply, Command } from '@redis/client/dist/lib/RESP/types';
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. I see a spec.ts

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit tests only..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to add an execution test, but not sure why I'm seeing output that I am via a decoder

i.e. via telnet

FT.EXPLAIN index '*'
$11
<WILDCARD>

via cli

127.0.0.1:6379> FT.EXPLAIN index '*'
"<WILDCARD>\n"
127.0.0.1:6379>

but via node-redis have to do this to make it pass

assert.equal('<WILDCARD>}\n', reply);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leibale thoughts?

Comment on lines 142 to 145
export interface SampleRawReply2 {
timestamp: NumberReply;
value: BlobStringReply;
};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an object/map in resp2 from the server? no way.. TuplesReply<[timestamp: NumberReply, value: BlobStringReply]>; is right i think

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copied from v4 badly I think. will do what you said.

@@ -18,14 +19,47 @@ export function pushFilterArgument(args: CommandArguments, filter: RedisVariadic
return pushVariadicArguments(args, filter);
}

export type MGetRawReply2 = Array<[
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use ArrayReply instead of Array

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done and BlobStringReply instead of string and UnwrapReply in transformReply

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

though had some issues with typing in the transformReply, needs to be validated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and same in RANGE/MRANGE/_WITHLABELS/

@@ -18,14 +19,47 @@ export function pushFilterArgument(args: CommandArguments, filter: RedisVariadic
return pushVariadicArguments(args, filter);
}

export type MGetRawReply2 = Array<[
key: string,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BlobStringReply?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it should be BlobStringReply, this was just a port over from v2, but it's a RedisModule_ReplyWithStringBuffer() so BlobString. Will fix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed above

};
},
3(reply: SampleRawReply[3]) {
const [timestamp, value] = reply as unknown as UnwrapReply<typeof reply>;
3(reply: SampleRawReply3): SampleReply3 {
Copy link
Collaborator

@leibale leibale Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In RESP3 the reply raw reply type should be

type SampleRawReply3 = ArrayReply<TuplesReply<[timestamp: NumberReply, value: DoubleReply]>>;

(I checked that in TS.RANGE, I guess its true for the rest of them?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I'm confused about how tuplereply and array reply work. do they have to be used together?

i.e. in looking at it now i see for range, we basically just have a single element but the function is a an arrayreply wrapping the tuple reply. I'm confused overall.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in looking at it more, I dont see why you would make SampleRawReply3 an ArrayReply, I'd think both resp2/resp3 should just be TupleReply and then RANGE et al would have it wrapped in an ArrayReply as get an array of the tuples. In practice then, resp2/resp3 would be the exact same then (at least for range) and therefore the code could be simplified.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type in my comments is for samples, but named sample, my bad..

This should work

type SampleRawReply = TuplesReply<[timestamp: NumberReply, value: DoubleReply]>;

const transformSampleReply = {
  2(reply: Resp2Reply<SampleRawReply>) {
    const [ timestamp, value ] = reply as unknown as UnwrapReply<typeof reply>;
    return {
      timestamp,
      value: Number(value)
    };
  },
  3(reply: SampleRawReply) {
    const [ timestamp, value ] = reply as unknown as UnwrapReply<typeof reply>;
    return {
      timestamp,
      value
    };
  }
};

type SamplesRawReply = ArrayReply<SampleRawReply>;

const transformSamplesReply = {
  2(reply: Resp2Reply<SamplesRawReply>) {
    return (reply as unknown as UnwrapReply<typeof reply>)
      .map(sample => transformSampleReply[2](sample));
  },
  3(reply: SamplesRawReply) {
    return (reply as unknown as UnwrapReply<typeof reply>)
      .map(sample => transformSampleReply[3](sample));  }
};

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tried implementing this, but cant do it right.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe got it now

@sjpotter sjpotter closed this Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants